Audio system

ABSTRACT

Methods and systems for providing audio information based on gaze information are described. In an example, a computing device may be configured to receive gaze information indicative of a wearer-view associated with the computing device. Based on the gaze information, the computing device may be configured to determine content displayed in the wearer-view. Further, the computing device may be configured to determine audio information relating to the content and cause an audio stream associated with the audio information to be played at the computing device.

BACKGROUND

Augmented reality generally refers to a real-time view of a real-worldenvironment that is augmented with additional content. A user mayexperience augmented reality through the use of a computing device. Thecomputing device may be configured to generate the real-time view of theenvironment, either by allowing a user to directly view the environmentor by allowing the user to indirectly view the environment by generatingand displaying a real-time representation of the environment to beviewed by the user.

The additional content may include, for example, a user-interfacethrough which the user may interact with the computing device. Thecomputing device may overlay the view of the environment with theuser-interface such that the user sees the view of the environment andthe user-interface at the same time.

SUMMARY

The present application discloses systems and methods for providingaudio information based on gaze information. In one aspect, a method isdescribed. The method may comprise receiving, at a computing device,gaze information indicative of a wearer-view associated with thecomputing device. The method also may comprise determining, based on thegaze information, a video advertisement in a physical advertisementspace in the wearer-view. The method further may comprise determining anaudio advertisement relating to the video advertisement. The method alsomay comprise causing the audio advertisement to be played at thecomputing device.

In still another aspect, a system is described. The system may comprisean eye-tracking system. The system also may comprise a computing devicein communication with the eye-tracking system. The computing device maybe configured to receive, from the eye-tracking system, gaze informationindicative of a wearer-view associated with the computing device. Thecomputing device also may be configured to determine, based on the gazeinformation, content depicted on a display device in the wearer-view.The computing device further may be configured to determine audioinformation relating to the content. The computing device also may beconfigured to cause an audio stream associated with the audioinformation to be played at the computing device.

In another aspect, a non-transitory computer readable medium havingstored thereon instructions executable by a computing device to causethe computing device to perform functions is described. The functionsmay comprise receiving, at a computing device, gaze informationindicative of a wearer-view associated with the computing device. Thefunctions also may comprise determining, based on the gaze information,a video advertisement in a physical advertisement space in thewearer-view. The functions further may comprise determining an audioadvertisement relating to the video advertisement. The function also maycomprise causing the audio advertisement to be played at the computingdevice.

The foregoing summary is illustrative only and is not intended to be inany way limiting. In addition to the illustrative aspects, embodiments,and features described above, further aspects, embodiments, and featureswill become apparent by reference to the figures and the followingdetailed description.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1A illustrates an example system for receiving, transmitting, anddisplaying data, in accordance with an embodiment.

FIG. 1B illustrates an alternate view of the system illustrated in FIG.1A, in accordance with an embodiment.

FIG. 2 illustrates another example system for receiving, transmitting,and displaying data, in accordance with an embodiment.

FIG. 3 illustrates another example system for receiving, transmitting,and displaying data, in accordance with an embodiment.

FIG. 4 is a flowchart of an example method for providing audioinformation based on gaze information, in accordance with an embodiment.

FIG. 5 illustrates an example implementation for providing audioinformation based on gaze information, in accordance with an embodiment.

FIG. 6 is a flowchart of another example method for providing audioinformation based on gaze information, in accordance with an embodiment.

FIG. 7 illustrates aspects of example implementations for providingaudio information based on gaze information, in accordance with anembodiment.

FIG. 8 is a functional block diagram illustrating an example computingdevice used in a computing system that is arranged in accordance with atleast some embodiments described herein.

FIG. 9 is a schematic illustrating a conceptual partial view of anexample computer program product that includes a computer program forexecuting a computer process on a computing device, arranged accordingto at least some embodiments presented herein.

DETAILED DESCRIPTION

The following detailed description describes various features andfunctions of the disclosed systems and methods with reference to theaccompanying figures. In the figures, similar symbols identify similarcomponents, unless context dictates otherwise. The illustrative systemand method embodiments described herein are not meant to be limiting. Itmay be readily understood that certain aspects of the disclosed systemsand methods can be arranged and combined in a wide variety of differentconfigurations, all of which are contemplated herein.

In an example, a computing device may be configured to receive gazeinformation indicative of a wearer-view associated with the computingdevice. Based on the gaze information, the computing device may beconfigured to determine a video advertisement in a physicaladvertisement space in the wearer-view. Further, the computing devicemay be configured to determine an audio advertisement relating to thevideo advertisement and cause the audio advertisement to be played atthe computing device.

In another example, the computing device may be configured to determine,based on the gaze information, content depicted on a display device inthe wearer-view. Also, the computing device may be configured todetermine audio information relating to the content and cause an audiostream associated with the audio information to be played at thecomputing device. The content may include one or more of a video, aninteractive media element, a game, and a graphical user interface, forexample. In an example, the content may be displayed as multiple itemson the display device, and each item of the multiple items may have anassociated audio stream. In another example, respective contents may bedisplayed on respective display devices, and the computing device may beconfigured, based on the gaze information indicating which respectivecontent is gazed at, to cause a given audio stream corresponding to therespective content to be played at the computing device.

In still another example, the computing device may cause an audio streamthat is thematically related to what a user is looking at orcircumstances of the user. As an example, the user may close his eyesand, accordingly, the computing device may cause a stream of soft musicto play at the computing device. In another example, the user may belooking at text (e.g., an article), and the computing device may stopplaying any audio streams. In still another example, the user may bedriving a car in traffic, and the computing device may cause white noisesound to play at the computing device to reduce perceived loudness ofthe traffic. Other examples are possible.

Referring now to the Figures, FIG. 1A illustrates an example system forreceiving, transmitting, and displaying data, in accordance with anembodiment. The system is shown as a wearable computing device in a formof a head-mounted device 102 as an example. Other types of wearablecomputing devices could additionally or alternatively be used. Asillustrated in FIG. 1A, the head-mounted device 102 may include frameelements including lens-frames 104, 106 and a center frame support 108,lens elements 110, 112, and extending side-arms 114, 116. The centerframe support 108 and the extending side-arms 114, 116 are configured tosecure the head-mounted device 102 to a user's face via a user's noseand ears, respectively.

Each of the frame elements 104, 106, and 108 and the extending side-arms114, 116 may be formed of a solid structure of plastic and/or metal, ormay be formed of a hollow structure of similar material so as to allowwiring and component interconnects to be internally routed through thehead-mounted device 102. Other materials are possible as well.

One or more of each of the lens elements 110, 112 may be formed of anymaterial that can suitably display a projected image or graphic (e.g., auser-interface). Each of the lens elements 110, 112 may also besufficiently transparent to allow a user to see through the lenselement. Combining these two features of the lens elements 110, 112 mayfacilitate an augmented reality or heads-up display where the projectedimage or graphic is superimposed over a real-world view as perceived bythe user through the lens elements 110, 112.

The extending side-arms 114, 116 may each be projections that extendaway from the lens-frames 104, 106, respectively, and may be positionedbehind a user's ears to secure the head-mounted device to the user. Insome examples, the extending side-arms 114, 116 may further secure thehead-mounted device 102 to the user by extending around a rear portionof the user's head. Additionally or alternatively, for example, thehead-mounted device 102 may connect to or be affixed within ahead-mounted helmet structure. Other possibilities exist as well.

The head-mounted device 102 may also include an on-board computingsystem 118, a video camera 120, at least one sensor 122, a speaker 123,a finger-operable touch pad 124, and a button 126. The on-boardcomputing system 118 is shown to be positioned on the extending side-arm114 of the head-mounted device 102; however, the on-board computingsystem 118 may be provided on other parts of the head-mounted device 102or may be positioned remote from the head-mounted device 102 (e.g., theon-board computing system 118 could be connected via a wired or wirelessconnection to the head-mounted device 102). The on-board computingsystem 118 may include a processor and data storage, for example, amongother components. The on-board computing system 118 may be configured toreceive and analyze data from the video camera 120, the at least onesensor 122, and the finger-operable touch pad 124 (and possibly fromother user-input devices, user-interfaces, or both) and generate imagesand graphics for output by the lens elements 110 and 112. The on-boardcomputing system 118 may additionally include a microphone for userinput (not shown).

The video camera 120 is shown positioned on the extending side-arm 114of the head-mounted device 102; however, the video camera 120 may beprovided on other parts of the head-mounted device 102. The video camera120 may be configured to capture images at various resolutions or atdifferent frame rates. Video cameras with a small form-factor, such asthose used in cell phones or webcams, for example, may be incorporatedinto an example of the head-mounted device 102.

Further, although FIG. 1A illustrates one video camera 120, more videocameras may be used, and each may be configured to capture the sameview, or to capture different views. For example, the video camera 120may be forward facing to capture at least a portion of the real-worldview perceived by the user. This forward facing image captured by thevideo camera 120 may then be used to generate an augmented reality whereimages and/or graphics appear to interact with the real-world viewperceived by the user.

The at least one sensor 122 is shown on the extending side-arm 116 ofthe head-mounted device 102; however, the at least one sensor 122 may bepositioned on other parts of the head-mounted device 102. The sensor atleast one 122 may include one or more movement sensors, such as one orboth of a gyroscope or an accelerometer, for example. Other sensingdevices may be included within, or in addition to, the at least onesensor 122, or other sensing functions may be performed by the at leastone sensor 122.

The speaker 123 is shown on the extending side-arm 114 of thehead-mounted device 102; however, the speaker 123 may be positioned onother parts of the head-mounted device 102. The on-board computingsystem 118 may be configured to cause an audio stream to be playedthrough the speaker 123, for example. Although one speaker 123 is shown,more than one speaker can be coupled to the head-mounted device 102.

The finger-operable touch pad 124 is shown on the extending side-arm 114of the head-mounted device 102; however, the finger-operable touch pad124 may be positioned on other parts of the head-mounted device 102.Also, more than one finger-operable touch pad may be present on thehead-mounted device 102. The finger-operable touch pad 124 may be usedby a user to input commands. The finger-operable touch pad 124 may senseat least one of a position and a movement of a finger via capacitivesensing, resistance sensing, or a surface acoustic wave process, amongother possibilities. The finger-operable touch pad 124 may be capable ofsensing finger movement in a direction parallel and/or planar to asurface of the finger-operable touch pad 124, in a direction normal tothe surface, or both, and may also be capable of sensing a level ofpressure applied to the pad surface. The finger-operable touch pad 124may be formed of one or more translucent or transparent insulatinglayers and one or more translucent or transparent conducting layers.Edges of the finger-operable touch pad 124 may be formed to have araised, indented, or roughened surface, so as to provide tactilefeedback to a user when the user's finger reaches the edge, or otherarea, of the finger-operable touch pad 124. If more than onefinger-operable touch pad is present, each finger-operable touch pad maybe operated independently, and may provide a different function. Forexample, the finger operable touch pad may include a button with variousfunctionalities.

The button 126 is shown on the extending arm 114 of the head-mounteddevice 102; however, the button 126 may be positioned on other parts ofthe head-mounted device 102. Also, more than one button such as thebutton 126 may be present on the head-mounted device 126.

FIG. 1B illustrates an alternate view of the system illustrated in FIG.1A, in accordance with an example. As shown in FIG. 1B, the lenselements 110, 112 may act as display elements. The head-mounted device102 may include a first projector 128 coupled to an inside surface ofthe extending side-arm 116 and configured to project a display 130 ontoan inside surface of the lens element 112. Additionally oralternatively, a second projector 132 may be coupled to an insidesurface of the extending side-arm 114 and configured to project adisplay 134 onto an inside surface of the lens element 110.

The lens elements 110, 112 may act as a combiner in a light projectionsystem. Further, in some examples, the lens elements 110, 112 mayinclude a coating that reflects the light projected onto them from theprojectors 128, 132.

In alternative examples, other types of display elements may also beused. For example, the lens elements 110, 112 themselves may include: atransparent or semi-transparent matrix display, such as anelectroluminescent display or a liquid crystal display, one or morewaveguides for delivering an image to the user's eyes, or other opticalelements capable of delivering an in focus near-to-eye image to theuser. A corresponding display driver may be disposed within the frameelements 104, 106 for driving such a matrix display. Alternatively oradditionally, a laser or light emitting diode (LED) source and scanningsystem could be used to draw a raster display directly onto the retinaof one or more of the user's eyes. In these examples, a reflectivecoating on the lenses 110, 112 may be omitted. Other possibilities existas well.

FIG. 2 illustrates another example system for receiving, transmitting,and displaying data, in accordance with an embodiment. The system isshown in the form of a wearable computing device 202. The wearablecomputing device 202 may include frame elements, side-arms, and lenselements, which may be similar to those described above in connectionwith FIGS. 1A and 1B. The wearable computing device 202 may additionallyinclude an on-board computing system 204 and a video camera 206, whichmay also be similar to those described above in connection with FIGS. 1Aand 1B. The video camera 206 is shown mounted on a frame of the wearablecomputing device 202; however, the video camera 206 may be mounted atother positions as well.

As shown in FIG. 2, the wearable computing device 202 may include asingle display 208 which may be coupled to the device. The display 208may be similar to the display described above in connection with FIGS.1A and 1B. The display 208 may be formed on one of the lens elements ofthe wearable computing device 202, and may be configured to overlayimages and/or graphics (e.g., a user-interface) on the user's view ofthe physical world. The display 208 is shown to be provided in a centerof a lens of the wearable computing device 202; however, the display 208may be provided in other positions. The display 208 is controllable viathe computing system 204 that is coupled to the display 208 via anoptical waveguide 210.

FIG. 3 illustrates another example system for receiving, transmitting,and displaying data, in accordance with an embodiment. The system isshown in the form of a wearable computing device 302. The wearablecomputing device 302 may include side-arms 304, a center frame support306, and a bridge portion with nosepiece 308. In the example shown inFIG. 3, the center frame support 306 connects the side-arms 304. Thewearable computing device 302 does not include lens-frames containinglens elements. The wearable computing device 302 may additionallyinclude an on-board computing system 310 and a video camera 312, whichmay be similar to those described above in connection with FIGS. 1A and1B. Additionally, the wearable computing device 302 may include a button314 that, for example, may include a multi-state button with multiplecorresponding functions.

The wearable computing device 302 may include a single lens element 316that may be coupled to one of the side-arms 304 or the center framesupport 306. The lens element 316 may include a display, which may besimilar to the display described above in connection with FIGS. 1A and1B, and may be configured to overlay images and/or graphics (e.g., auser-interface, menus, etc.) upon the user's view of the physical world.In one example, the single lens element 316 may be coupled to a side ofthe extending side-arm 304. The single lens element 316 may bepositioned in front of or proximate to a user's eye when the wearablecomputing device 302 is worn by a user. For example, the single lenselement 316 may be positioned below the center frame support 306, asshown in FIG. 3.

In some examples, a wearable computing device (such as any of thewearable computing devices 102, 202, and 302 described above) may beconfigured to operate in a computer network structure. To this end, thewearable computing device may be configured to connect to one or moreremote devices using a communication link or links.

The remote device(s) may be any type of computing device or transmitter,such as, for example, a laptop computer, a mobile telephone, or tabletcomputing device, etc., that is configured to transmit data to thewearable computing device. The wearable computing device may beconfigured to receive the data and, in some cases, provide a displaythat is based at least in part on the data.

The remote device(s) and the wearable computing device may each includehardware to enable the communication link(s), such as processors,transmitters, receivers, antennas, etc. The communication link(s) may bea wired or a wireless connection. For example, the communication linkmay be a wired serial bus, such as a universal serial bus or a parallelbus, among other connections. As another example, the communication linkmay be a wireless connection using, e.g., Bluetooth® radio technology,communication protocols described in IEEE 802.11 (including any IEEE802.11 revisions), Cellular technology (such as GSM, CDMA, UMTS, EV-DO,WiMAX, or LTE), or Zigbee® technology, among other possibilities. Eitherof such a wired and/or wireless connection may be a proprietaryconnection as well. The remote device(s) may be accessible via theInternet and may include a computing cluster associated with aparticular web service (e.g., social-networking, photo sharing, addressbook, etc.).

As described above in connection with FIGS. 1-3, an example wearablecomputing device may include, or may otherwise be communicativelycoupled to, a computing system, such as computing system 118, computingsystem 204, or computing system 310.

FIG. 4 is a flow chart of an example method 400 for providing audioinformation based on gaze information, in accordance with an embodiment.FIG. 5 illustrates an example implementation for providing audioinformation based on gaze information, in accordance with an embodiment,to illustrate the method 400. FIGS. 4 and 5 will be described together.

The method 400 may include one or more operations, functions, or actionsas illustrated by one or more of blocks 402-408. Although the blocks areillustrated in a sequential order, these blocks may in some instances beperformed in parallel, and/or in a different order than those describedherein. Also, the various blocks may be combined into fewer blocks,divided into additional blocks, and/or removed based upon the desiredimplementation

In addition, for the method 400 and other processes and methodsdisclosed herein, the flowchart shows functionality and operation of onepossible implementation of present examples. In this regard, each blockmay represent a module, a segment, or a portion of program code, whichincludes one or more instructions executable by a processor forimplementing specific logical functions or steps in the process. Theprogram code may be stored on any type of computer readable medium ormemory, for example, such as a storage device including a disk or harddrive. The computer readable medium may include a non-transitorycomputer readable medium, for example, such as computer-readable mediathat stores data for short periods of time like register memory,processor cache and Random Access Memory (RAM). The computer readablemedium may also include non-transitory media or memory, such assecondary or persistent long term storage, like read only memory (ROM),optical or magnetic disks, compact-disc read only memory (CD-ROM), forexample. The computer readable media may also be any other volatile ornon-volatile storage systems. The computer readable medium may beconsidered a computer readable storage medium, a tangible storagedevice, or other article of manufacture, for example.

In addition, for the method 400 and other processes and methodsdisclosed herein, each block in FIG. 4 may represent circuitry that iswired to perform the specific logical functions in the process.

At block 402, the method 400 includes receiving, at a computing device,gaze information indicative of a wearer-view associated with thecomputing device. In an example, the computing device may include awearable computing device, such as the wearable computing devicesdescribed in FIGS. 1A-3. In another example, the computing device mayinclude a server or other remote device (e.g., mobile phone) incommunication with the wearable computing device. In the descriptionherein, the computing device may refer to the wearable computing deviceor the server in communication with the wearable computing device.

In an example, the computing device may be in communication with aneye-tracking system. The eye-tracking system may include hardware suchas an infrared camera and at least one infrared light source. Theinfrared camera may be utilized by the eye-tracking system to captureimages of an eye of a user. The images may include either video imagesor still images or both. The images obtained by the infrared cameraregarding the eye of the user may help determine where the user may belooking within a given field of view, for instance, by ascertaining alocation of an eye pupil of the user. As another example, theeye-tracking system may include a low power reflectivity sensor systemthat detects where the eye pupil may be pointing or gazing. The lowpower reflectivity sensor system may include an infrared (IR) lightemitting diode (LED) and photo detector that may be directed at the eyepupil. When the eye pupil gazes at the IR LED, the amount of IR lightreflected back to the photo detector may drop, for example. Thecomputing device, thus, may be configured to receive gaze informationfrom the eye-tracking system and to make a determination regarding adirection of a gaze of the user, also termed a gaze axis. The computingdevice, thus, may be configured to determine a wearer-view based on thegaze information; and further may be configured to cause animage-capture device (e.g., a camera) coupled to the computing device tocapture an image or a video of the determined wearer-view.

In the example where the computing device includes the wearablecomputing device, the image or the video may be captured from apoint-of-view (POV) of the wearable computing device by a forward-facingcamera coupled to the wearable computing device, for example. In oneexample, the forward-facing camera may be coupled to a head-mounteddisplay (HMD) coupled to the wearable computing device. In this example,the image or the video captured by the forward-facing camera may providean indication of what the user of the wearable computing device or theHMD may be looking at.

At block 404, the method 400 includes determining, based on the gazeinformation, a video advertisement in a physical advertisement space inthe wearer-view. FIG. 5 illustrates a user 502 wearing a wearablecomputing device 504 (e.g., any of the wearable computing devicesdescribed in FIGS. 1A-3). The wearable computing device 504 may beconfigured to receive gaze information indicative of a wearer-view 506associated with the wearable computing device 504. The gaze information,for example, may include an image or a video of the wearer-view 506,captured by an image-capture device coupled to the wearable computingdevice 504. As an example, the image-capture device may include a camerasuch as the camera 120 in FIG. 1A, or the camera 206 in FIG. 2, or thecamera 312 in FIG. 3 or any other image-capture device coupled to arespective wearable computing device.

In an example, based on the gaze information (e.g., the image or thevideo), the wearable computing device 504 may be configured to determinethat the wearer-view 506 includes a video advertisement 508 displayed ina physical advertisement space 510. The physical advertisement space510, for example, may include a billboard, an advertising screen in amall, or any other configuration of physical advertising space where agiven video advertisement is displayed.

In one example, the wearable computing device 504 may be configured touse image recognition techniques known in the art to identify the videoadvertisement 508 based on the gaze information (e.g., determine thevideo advertisement 508 depicted in the image or the video). In anotherexample, the wearable computing device may be configured to communicatethe gaze information through a network 512 (wired or wireless) to aserver 514. In this example, the server 514 may be configured to receivethe gaze information and may be configured to determine or identify thevideo advertisement 508 in the physical advertisement space 510, basedon the gaze information.

In still another example, the wearable computing device 504 may beconfigured to process the gaze information and generate a visual searchquery based on the video advertisement 508. The visual search query, forexample, may include the image or a query image extracted from the imageor a query video extracted from the video included in the gazeinformation in addition to metadata associated with the videoadvertisement 508 (e.g., features, colors, categorization, product orentity advertised, etc.). The wearable computing device 504 may beconfigured to communicate the visual search query through the network512 to the server 514; and, in response to receiving the visual searchquery, the server 514 may be configured to recognize or identify thevideo advertisement 508. As an example, the server 514 may be configuredto access or be in communication with an advertisement database 516 thatmay include a repository of video advertisements. The advertisementdatabase 516, for example, may index reference advertisements to comparethe received gaze information and/or visual search query with storedinformation so as to identify the video advertisement 508. The server514, thus, may be configured to communicate with the advertisementdatabase 516, based on the gaze information and/or the visual searchquery, to determine the associated video advertisement 508.

Referring back to FIG. 4, at block 406, the method 400 includesdetermining an audio advertisement relating to the video advertisement.The video advertisement may be a portion of a media advertisement thatalso includes a corresponding audio advertisement. As an example, thevideo advertisement may include a video of a vehicle being advertisedshowing exterior and interior details of the vehicle; and, in thisexample, the audio advertisement may include an audio commentary thatexplains or defines the details in the video advertisement. Thecomputing device may be configured to determine the audio advertisementrelating to the determined video advertisement.

For example, in FIG. 5, the wearable computing device 504, uponreceiving the gaze information and determining the video advertisement508, may be configured to determine the audio advertisementcorresponding to the video advertisement 508. In another example, theserver 514, upon receiving the gaze information or the visual searchquery and determining the video advertisement 508, may be configured,through access or communication with the advertisement database 516, todetermine the audio advertisement corresponding to the videoadvertisement 508.

Referring back to FIG. 4, at block 408, the method 400 includes causingthe audio advertisement to be played at the computing device. Referringto FIG. 5, either the wearable computing device 504 or the server 514may be configured to cause the audio advertisement to be played at thewearable computing device 504. In one example, the wearable computingdevice 504 may be configured to receive an audio file or an audio streamof the audio advertisement from the server 514 through the network 512;and may be configured to cause a software application to play the audiostream by executing instructions stored on a memory coupled to thewearable computing device 504, for example. In this example, thewearable computing device 504 may be configured to cause the audiostream to play through one or more speakers (e.g., the speaker 123illustrated in FIG. 1B) that may be coupled to the wearable computingdevice 504.

In one example, the audio advertisement may be synchronized to the videoadvertisement 508. For example, the server 514 may be configured to sendthe audio advertisement and a starting point or starting time in theaudio advertisement to start at which in order to be in synchrony withthe video advertisement 508. In another example, the audio advertisementmay not be synchronized to the video advertisement 508 and may startplaying at a beginning of the audio advertisement. In still anotherexample, the audio advertisement determined and caused to be played bythe computing device may be a portion of a longer audio advertisement.

In an example, the computing device (i.e., either the wearable computingdevice 504 or the server 514) may be configured to determine, based onthe gaze information, an amount of time during which the videoadvertisement 508 is gazed at. The computing device may be configured tocompare the amount of time to a predetermined threshold time period andmay be configured to cause the audio advertisement to be played at thecomputing device in response to determining that the amount of timeexceeds the predetermined threshold time period, which may be indicativeof the user 502 heeding or paying attention to the video advertisement508 and being interested in hearing the audio advertisement relating tothe video advertisement 508. In another example, the computing devicemay be configured to determine a focus metric, which may be a functionof the amount of time during which the video advertisement is gazed at.The focus metric may be indicative of a level of focus or interest ofthe user 502 in the video advertisement 508. The computing device may beconfigured to cause the audio advertisement to be played at thecomputing device based on a comparison of the focus metric to apredetermined focus threshold.

In some examples, the computing device may be configured to determine,based on the gaze information, a change in the wearer-view 506. Forexample, the gaze information may indicate that the user 502 disregarded(e.g., looked away from) the video advertisement 508. Accordingly, thecomputing device may be configured to stop playing the audioadvertisement. Further, based on the gaze information, the computingdevice may be configured to determine that the user 502 re-focused onthe video advertisement 508 and, accordingly, may be configured to causethe audio advertisement to resume playing at the computing device.

In the examples described above, the video advertisement 508 is shown inthe physical advertisement space 510, which can be a billboard or anyscreen that the user 502 may be looking at. Alternatively oradditionally, the computing device may be configured to cause a videoportion of an advertisement to be displayed in an advertisement space ona head-mounted display (HMD) coupled to the computing device. Thecomputing device may be configured to receive gaze information from aneye-tracking system coupled to the computing device and determine thatthat the video portion is gazed at. Further, the computing device may beconfigured to cause an audio portion of the advertisement to be playedat the computing device.

FIG. 6 is a flowchart of another example method for providing audioinformation based on gaze information, in accordance with an embodiment.FIG. 7 illustrates aspects of example implementations for providingaudio information based on gaze information, in accordance with anembodiment, to illustrate the method 600. FIGS. 6 and 7 will bedescribed together.

The method 600 may include one or more operations, functions, or actionsas illustrated by one or more of blocks 602-608. Although the blocks areillustrated in a sequential order, these blocks may in some instances beperformed in parallel, and/or in a different order than those describedherein. Also, the various blocks may be combined into fewer blocks,divided into additional blocks, and/or removed based upon the desiredimplementation.

At block 602, the method 600 includes receiving at a computing device,from an eye-tracking system, gaze information indicative of awearer-view associated with the computing device. In an example, thecomputing device may include a wearable computing device, such as thewearable computing devices described in FIGS. 1A-3. In another example,the computing device may include a server in communication with thewearable computing device. In the description herein, the computingdevice may refer to the wearable computing device or the server incommunication with the wearable computing device.

The computing device may be coupled to an eye-tracking system. Thecomputing device may be configured to determine a wearer-view based ongaze information received from the eye-tracking system and may beconfigured to cause an image-capture device coupled to the computingdevice to capture an image or a video of the determined wearer-view.

In the example where the computing device includes the wearablecomputing device, the image or the video may be captured from apoint-of-view (POV) of the wearable computing device by a forward-facingcamera coupled to the wearable computing device. In one example, theforward-facing camera may be coupled to a head-mounted display (HMD)coupled to the wearable computing device. In this example, the image orthe video captured by the forward-facing camera may provide anindication of what a user of the wearable computing device or the HMDmay be looking at or tracking with the user's eyes.

FIG. 7 illustrates a user 702 wearing a wearable computing device 704.The wearable computing device 704 may be configured to receive gazeinformation indicative of a wearer-view associated with the wearablecomputing device 704. The gaze information, for example, may include animage or a video captured by an image-capture device coupled to thewearable computing device 704. As an example, the image-capture devicemay include the camera 120 in FIG. 1A, or the camera 206 in FIG. 2, orthe camera 312 in FIG. 3 or any other image-capture device coupled to arespective wearable computing device.

Referring back to FIG. 6, at block 604, the method 600 includesdetermining, based on the gaze information, content depicted on adisplay device in the wearer-view. For example, in FIG. 7, based on thegaze information (e.g., the image or the video), the wearable computingdevice 704 may be configured to determine that the wearer-view includescontent depicted on a display device in the wearer-view. The displaydevice may include any type of display devices depending on anenvironment of the user 702. In FIG. 7, different types of displaydevices are illustrated including an advertisement screen 706, anoverhead projector screen 708 having multiple content items 709A-Cprojected on the overhead projector screen 708, a terminal screen 710, alaptop 712 used by an operator 713, a personal digital assistant (PDA)714, an appliance 716 including a display device 717, a personalcomputer 718, and a stack of screens (e.g., televisions) 720. Thesetypes are shown for illustration only. Other types of display deviceswith displayed content can be used. All or a subset of these displaydevices may exist at a given location where the user 702 may be.

The wearable computing device 704 may be configured to determine, basedon the gaze information, the wearer-view (e.g., where the user 702 maybe gazing at) and determine respective content displayed on a respectivedisplay device in the determined wearer-view.

In one example, the wearable computing device 704 may be configured useimage recognition techniques known in the art to determine therespective content depicted on the respective display device. In anotherexample, the wearable computing device 704 may be configured tocommunicate the gaze information and/or a visual search query associatedwith the respective content through a network (e.g., the internet) to aserver. In this example, the server may be configured to receive thegaze information (e.g., the image or the video) and/or the visual searchquery and may be configured to determine the respective content.

The content may include, for example, a video, an interactive mediaelement (e.g., a webpage or an animation), a game, an advertisement, agraphic user-interface (e.g., user-interface of a software for playingmusic), or any other type of content.

Referring back to FIG. 6, at block 606, the method 600 includesdetermining audio information relating to the content. For example, inFIG. 7, the wearable computing device 704, upon receiving the gazeinformation and determining the content, may be configured to determineaudio information (e.g., audio stream) corresponding to the contentdisplayed. In another example, the server in communication with thewearable computing device 704, upon receiving the gaze information orthe visual search query and determining the content, may be configured,through access or communication with a database, to determine the audioinformation corresponding to the content displayed. For example, if thecontent includes a game displayed on the laptop 712 in FIG. 7, the audioinformation may include voice commentary that explains the game or musicthat accompanies the game. In this example, the operator 713 may have achoice to register the laptop 712 to opt in or out to allow or preventthe user 702 from accessing the audio information.

In another example, the content may be programming instructionsdisplayed on the appliance 716, and the audio information may includemusic or audio instructions that accompany the programming instructions.In still another example, the content may include a graphicuser-interface of software application for playing music, and the audioinformation may include the music or a song playing at the displaydevice in the wearer view. These examples are for illustrations only.Other examples of content displayed on a given display device having anaccompanying audio are possible.

In addition, the computing device may be configured to determine otherrelevant audio information such as a timestamp indicative of a currenttime position in the content displayed and a time position of anaccompanying audio stream corresponding to the current time position inthe content, for example.

Referring back to FIG. 6, at block 608, the method 600 includes causingan audio stream associated with the audio information to be played atthe computing device. As an example, one or more speakers (e.g., thespeaker 123 illustrated in FIG. 1B) may be coupled to the computingdevice, and the computing device may be configured to cause the audiostream to be played through the speakers.

In FIG. 7, either the wearable computing device 704 or the server incommunication with the wearable computing device 704 may be configuredto cause the audio stream to be played at the wearable computing device704. In one example, the wearable computing device 704 may be configuredto receive an audio file or an audio stream from the server through thenetwork and may be configured to cause a software application configuredto play the audio file by executing instructions stored on a memorycoupled to the wearable computing device 704, for example.

In an example, the audio stream may be synchronized to the content. Forexample, the computing device may be configured to cause the audiostream to be played starting at the time position (i.e., the timeposition determined, as part of the audio information, to correspond tothe current time position in the content displayed) to be in synchronywith the content. In another example, the audio stream may not besynchronized to the content and may start playing at a beginning of theaudio stream. In still another example, the audio stream determined andcaused to be played by the computing device may be a portion of a longeraudio stream.

To determine which display device or content is gazed at or focused onand cause a corresponding audio stream to be played, the computingdevice (i.e., either the wearable computing device 704 or the server incommunication with the wearable computing device 704) may be configuredto determine, based on the gaze information, a focus metric that may bea function of an amount of time during which the content is gaze at. Thecomputing device may be configured to cause the audio stream to beplayed at the computing device based on a comparison of the focus metricto a predetermined focus threshold.

In an example, the computing device (i.e., either the wearable computingdevice 704 or the server in communication with the wearable computingdevice 704) may be configured to determine, based on the gazeinformation, a change in the wearer-view. For example, the gazeinformation may indicate that the user 702 disregarded (e.g., lookedaway from) a respective content displayed on a respective display deviceof the display devices 706, 708, 710, 712, 714, 716, 718, and 720.Accordingly, the computing device may be configured to stop playing theaudio stream. Further, based on the gaze information, the computingdevice may be configured to determine that the user 702 re-focused onthe respective content and, accordingly, may be configured to cause theaudio stream to resume playing at the computing device. The audio streammay or may not be caused to resume playing in synchrony with therespective content.

In another example, the computing device may be configured to determine,based on the gaze information, a change in the wearer-view indicatingthat a first content on a first display device (e.g., the terminalscreen 710) is disregarded and that a second content displayed on asecond display device (e.g., the PDA 714) is focused on. Accordingly,the computing device may be configured to cause a first audio streamassociated with the first content to stop playing and cause a secondaudio stream relating to the second content displayed on the seconddisplay device to be played.

In still another example, the wearer-view may include multiple displaydevices. In this example, the computing device may be configured tochoose a respective display device and respective content on which theuser 702 gazes the longest. In another example, the user 702 may walkinto a room including multiple display devices and may scan around theroom (e.g., through head motion) to capture the display devices within anumber of wearer-views. In this example, the computing device may beconfigured to identify the display devices and provide a listing of thedisplay devices to the user 702 and cause an audio stream to play basedon receiving a selection of one of the display devices or respectivedisplay contents from the user 702.

In yet another example, the content may comprise multiple itemsdisplayed on the display device. For example, multiple content itemsincluding a video 709A, a graphic user-interface 709B, and a game 709Cmay be displayed on the overhead projector screen 708 (or any otherdisplay device). The computing device (i.e., either the wearablecomputing device 704 or the server in communication with the wearablecomputing device 704) may be configured to determine, based on the gazeinformation, that a first content item of the multiple items 709A-C isgaze at and cause an associated first audio stream to be played at thewearable computing device 704. Further, the computing device may beconfigured to determine, based on the gaze information, a change in thewearer-view indicating that the first content item is disregarded andthat a second content item is gazed at. Accordingly, the computingdevice may be configured to cause a second audio stream associated withthe second content item to be played at the computing device. Similarly,with respect to the stack of screens 720, the computing device may beconfigured to determine which screen of the stack of screens 720 isgazed at and cause an associated audio stream to be played at thewearable computing device 704.

In yet still another example, the computing device (i.e., either thewearable computing device 704 or the server in communication with thewearable computing device 704) may be configured to determine a type ofa given display device (e.g., through image recognition, or manufacturername on the device, etc.) to further help determine types of contentthat the given display device can display. For example, a televisionscreen (e.g., of the stack of screens 720) may display a televisionseries, movie or commercial but may not display a software application.However, content displayed on the PDA 714, for example, may include asoftware application.

In these examples, operators or owners of the display devices will havea choice to register the respective display device to opt in or out toallow or prevent the user 702 from accessing the audio information.

In the examples described above, the content is displayed on displaydevices away from the user 702. In an example, the wearable computingdevice 704 may be configured to generate a display of multiple contentitems on a head-mounted display (HMD) coupled to the wearable computingdevice 704. The wearable computing device 704 may be configured toreceive gaze information from an eye-tracking system coupled to thewearable computing device 704 and determine which content item of themultiple content items is gazed at (e.g., through communication with theserver). Further, the wearable computing device 704 may be configured tocause an associated audio stream to be played at the wearable computingdevice 704. The associated audio stream may or may not be synchronizedto the content item. The computing device also may be configured to stopplaying the associated audio stream upon receiving gaze informationindicating that the user 702 may be gazing at another content item andmay be configured to cause another audio stream corresponding to theother content item to be played instead.

FIG. 8 is a functional block diagram illustrating an example computingdevice 800 used in a computing system that is arranged in accordancewith at least some embodiments described herein. The computing devicemay be a personal computer, mobile device, cellular phone, video gamesystem, or global positioning system, and may be implemented as a clientdevice, a server, a system, a combination thereof, or may be part of thewearable computing systems or head-mounted devices shown in FIGS. 1-3.Alternatively, the computing device 800 may be communicatively coupledto the head-mounted devices via a wired or wireless connection.

In a basic configuration 802, computing device 800 may include one ormore processors 810 and system memory 820. A memory bus 830 can be usedfor communicating between the processor 810 and the system memory 820.Depending on the desired configuration, processor 810 can be of any typeincluding but not limited to a microprocessor (μP), a microcontroller(μC), a digital signal processor (DSP), or any combination thereof. Amemory controller 815 can also be used with the processor 810, or insome implementations, the memory controller 815 can be an internal partof the processor 810.

Depending on the desired configuration, the system memory 820 can be ofany type including but not limited to volatile memory (such as RAM),non-volatile memory (such as ROM, flash memory, etc.) or any combinationthereof. System memory 820 may include one or more applications 822, andprogram data 824. Application 822 may include audio informationalgorithm 823 that is arranged to provide inputs to the electroniccircuits, in accordance with the present disclosure. Program Data 824may include content information 825 that could be directed to any numberof types of data. In some examples, application 822 can be arranged tooperate with program data 824 on an operating system.

Computing device 800 can have additional features or functionality, andadditional interfaces to facilitate communications between the basicconfiguration 802 and any devices and interfaces. For example, datastorage devices 840 can be provided including removable storage devices842, non-removable storage devices 844, or a combination thereof.Examples of removable storage and non-removable storage devices includemagnetic disk devices such as flexible disk drives and hard-disk drives(HDD), optical disk drives such as compact disk (CD) drives or digitalversatile disk (DVD) drives, solid state drives (SSD), and tape drivesto name a few. Computer storage media can include volatile andnonvolatile, non-transitory, removable and non-removable mediaimplemented in any method or technology for storage of information, suchas computer readable instructions, data structures, program modules, orother data.

System memory 820 and storage devices 840 are examples of computerstorage media. Computer storage media includes, but is not limited to,RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM,digital versatile disks (DVD) or other optical storage, magneticcassettes, magnetic tape, magnetic disk storage or other magneticstorage devices, or any other medium which can be used to store thedesired information and which can be accessed by the computing device800. Any such computer storage media can be part of the computing device800.

The computing device 800 can also include output interfaces 850 that mayinclude a graphics processing unit 852, which can be configured tocommunicate to various external devices such as display devices 860 orspeakers via one or more A/V ports 854 or a communication interface 870.The communication interface 870 may include a network controller 872,which can be arranged to facilitate communications with one or moreother computing devices 880 and one or more sensors 882 over a networkcommunication via one or more communication ports 874. The one or moresensors 882 are shown external to the computing device 800, but may alsobe internal to the device. Communication media may be embodied bycomputer readable instructions, data structures, program modules, orother data in a modulated data signal, such as a carrier wave or othertransport mechanism, and includes any information delivery media. Amodulated data signal can be a signal that has one or more of itscharacteristics set or changed in such a manner as to encode informationin the signal. By way of example, and not limitation, communicationmedia can include wired media such as a wired network or direct-wiredconnection, and wireless media such as acoustic, radio frequency (RF),infrared (IR) and other wireless media.

In some embodiments, the disclosed methods may be implemented ascomputer program instructions encoded on a computer-readable storagemedia in a machine-readable format, or on other non-transitory media orarticles of manufacture. FIG. 9 is a schematic illustrating a conceptualpartial view of an example computer program product 900 that includes acomputer program for executing a computer process on a computing device,arranged according to at least some embodiments presented herein. In oneembodiment, the example computer program product 900 is provided using asignal bearing medium 901. The signal bearing medium 901 may include oneor more program instructions 902 that, when executed by one or moreprocessors may provide functionality or portions of the functionalitydescribed above with respect to FIGS. 1-8. Thus, for example, referringto the embodiments shown in FIGS. 4 and 6, one or more features ofblocks 402-408 and blocks 602-608 may be undertaken by one or moreinstructions associated with the signal bearing medium 901. In addition,the program instructions 902 in FIG. 9 describe example instructions aswell.

In some examples, the signal bearing medium 901 may encompass acomputer-readable medium 903, such as, but not limited to, a hard diskdrive, a Compact Disc (CD), a Digital Video Disk (DVD), a digital tape,memory, etc. In some implementations, the signal bearing medium 901 mayencompass a computer recordable medium 904, such as, but not limited to,memory, read/write (R/W) CDs, R/W DVDs, etc. In some implementations,the signal bearing medium 901 may encompass a communications medium 905,such as, but not limited to, a digital and/or an analog communicationmedium (e.g., a fiber optic cable, a waveguide, a wired communicationslink, a wireless communication link, etc.). Thus, for example, thesignal bearing medium 901 may be conveyed by a wireless form of thecommunications medium 905 (e.g., a wireless communications mediumconforming to the IEEE 802.11 standard or other transmission protocol).

The one or more programming instructions 902 may be, for example,computer executable and/or logic implemented instructions. In someexamples, a computing device such as the computing device 800 of FIG. 8may be configured to provide various operations, functions, or actionsin response to the programming instructions 902 conveyed to thecomputing device 800 by one or more of the computer readable medium 903,the computer recordable medium 904, and/or the communications medium905. It should be understood that arrangements described herein are forpurposes of example only. As such, those skilled in the art willappreciate that other arrangements and other elements (e.g. machines,interfaces, functions, orders, and groupings of functions, etc.) can beused instead, and some elements may be omitted altogether according tothe desired results. Further, many of the elements that are describedare functional entities that may be implemented as discrete ordistributed components or in conjunction with other components, in anysuitable combination and location.

While various aspects and embodiments have been disclosed herein, otheraspects and embodiments will be apparent to those skilled in the art.The various aspects and embodiments disclosed herein are for purposes ofillustration and are not intended to be limiting, with the true scopebeing indicated by the following claims, along with the full scope ofequivalents to which such claims are entitled. It is also to beunderstood that the terminology used herein is for the purpose ofdescribing particular embodiments only, and is not intended to belimiting.

What is claimed is:
 1. A method, comprising: receiving, at a wearablecomputing device, gaze information comprising: (i) an image or a videoindicative of a wearer-view through a field of view of the wearablecomputing device, and (ii) data indicating a view direction associatedwith the wearer-view; identifying a video advertisement that is playing,in a physical advertisement space captured in the image or the video, ata location in the image or the video that corresponds to the viewdirection, wherein identifying the video advertisement comprises:generating a search query based on (i) a portion of the image or thevideo at the location that corresponds to the view direction, and (ii)an amount of time during which the location in the image or the videocorresponds to the view direction, providing the search query to aserver, and receiving an identification of the video advertisement; inresponse to identifying the video advertisement that is playing in thephysical advertisement space that corresponds to the view direction,determining an audio advertisement relating to the video advertisement;and causing the audio advertisement to be played at the wearablecomputing device.
 2. The method of claim 1, wherein causing the audioadvertisement to be played at the wearable computing device comprisescausing the audio advertisement to be played at the wearable computingdevice in response to determining that the amount of time exceeds apredetermined threshold time period.
 3. The method of claim 1, furthercomprising: determining, based on the gaze information, a change in theview direction, wherein the change is indicative that the videoadvertisement is disregarded; causing the audio advertisement to stopplaying; and causing the audio advertisement to resume playing inresponse to determining that the view direction has changed back tocorrespond to the location at which video advertisement is playing. 4.The method of claim 1, wherein receiving the gaze information comprisesreceiving information relating to a gaze axis from an eye-trackingsystem coupled to the wearable computing device, wherein the dataindicating the view direction is based on the gaze axis.
 5. The methodof claim 4, further comprising determining the wearer-view based on thegaze axis.
 6. The method of claim 1, wherein the audio advertisement issynchronized to the video advertisement playing in the physicaladvertisement space.
 7. A system, comprising: an eye-tracking system; acamera coupled to a head-mounted display (HMD), wherein the camera isconfigured to capture a field of view of the HMD; and a computing devicein communication with the eye-tracking system and the camera, whereinthe computing device is configured to: receive, from the eye-trackingsystem, gaze information comprising: (i) an image or a video indicativeof a wearer-view through a field of view of the wearable computingdevice, and (ii) data indicating a view direction associated with thewearer-view; identify an animated content that is playing, on a displaydevice captured in the image or the video, at a location in the image orthe video that corresponds to the view direction, wherein, to identifythe animated content, the computing device is configured to: generate asearch query based on (i) a portion of the image or the video at thelocation that corresponds to the view direction, and (ii) an amount oftime during which the location in the image or the video corresponds tothe view direction, provide the search query to a server, and receive anidentification of the animated content; in response to identifying theanimated content that is playing on the display device that correspondsto the view direction, determine audio information relating to theanimated content; and cause an audio stream associated with the audioinformation to be played at the computing device.
 8. The system of claim7, wherein the computing device is further configured to: determine,based on the gaze information, a change in the view direction, whereinthe change is indicative of the animated content playing on the displaydevice being disregarded; and cause the audio stream to stop playing. 9.The system of claim 8, wherein the computing device is furtherconfigured to cause the audio stream to resume playing in response todetermining that the view direction has changed back to correspond tothe location at which the animated content is playing.
 10. The system ofclaim 7, wherein the eye-tracking system comprises a given camera,wherein the gaze information comprises one or more images captured bythe given camera, and wherein the computing device is further configuredto: determine a gaze axis based on the one or more images; and determinethe view direction based on the determined gaze axis.
 11. The system ofclaim 10, wherein the image or the video are captured for the determinedwearer-view.
 12. The system of claim 7, wherein the audio stream issynchronized to the animated content.
 13. The system of claim 7, whereinthe animated content comprises multiple items displayed on the displaydevice, and wherein the computing device is further configured to:identify, based on the gaze information, a respective item of themultiple items; and cause a respective audio stream relating to therespective item to be played at the computing device.
 14. The system ofclaim 13, wherein the computing device is further configured to:determine, based on the gaze information, a change in the viewdirection, wherein the change is indicative that the respective item isdisregarded; cause the respective audio stream item to stop playing;identify, based on the gaze information, a given item gazed at in thechanged view direction; and cause a given audio stream relating to thegiven item to be played at the computing device.
 15. The system of claim7, wherein the computing device is further configured to: determine,based on the gaze information, a change in the view direction, whereinthe change is indicative that the animated content on the display deviceis disregarded; cause the audio stream to stop playing; identify, basedon the gaze information, a respective content depicted on a respectivedisplay device in the changed view direction; and cause a respectiveaudio stream to be played at the computing device.
 16. The system ofclaim 7, wherein the animated content comprises one or more of a video,an interactive media element, a game, an advertisement, and a graphicaluser-interface.
 17. A non-transitory computer readable medium havingstored thereon instructions executable by a wearable computing device tocause the wearable computing device to perform functions comprising:receiving gaze information comprising: (i) an image or a videoindicative of a wearer-view through a field of view of the wearablecomputing device, and (ii) data indicating a view direction associatedwith the wearer-view; identifying a video advertisement that is playing,in a physical advertisement space captured in the image or the video, ata location in the image or the video that corresponds to the viewdirection, wherein identifying the video advertisement comprises:generating a search query based on (i) a portion of the image or thevideo at the location that corresponds to the view direction, and (ii)an amount of time during which the location in the image or the videocorresponds to the view direction, providing the search query to aserver, and receiving an identification of the video advertisement; inresponse to identifying the video advertisement that is playing in thephysical advertisement space that corresponds to the view direction,determining an audio advertisement relating to the video advertisement;and causing the audio advertisement to be played at the wearablecomputing device.
 18. The non-transitory computer readable medium ofclaim 17, wherein the instructions are further executable by thewearable computing device to cause the wearable computing device toperform functions comprising: determining, based on the gazeinformation, a focus metric, wherein the focus metric is based on theamount of time during which the location in the image or the videocorresponds to the view direction; and causing the audio advertisementto be played at the wearable computing device based on a comparison ofthe focus metric to a predetermined focus threshold.